Optuna: A hyperparameter optimization framework

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.

Key Features

Optuna has modern functionalities as follows:

Lightweight, versatile, and platform agnostic architecture
- Handle a wide variety of tasks with a simple installation that has few requirements.
Pythonic search spaces
- Define search spaces using familiar Python syntax including conditionals and loops.
Efficient optimization algorithms
- Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials.
Easy parallelization
- Scale studies to tens or hundreds of workers with little or no changes to the code.
Quick visualization
- Inspect optimization histories from a variety of plotting functions.

Basic Concepts

We use the terms study and trial as follows:

Study: optimization based on an objective function
Trial: a single execution of the objective function

Please refer to sample code below. The goal of a study is to find out the optimal set of hyperparameter values (e.g., classifier and svm_c) through multiple trials (e.g., n_trials=100). Optuna is a framework designed for the automation and the acceleration of the optimization studies.

import ...

# Define an objective function to be minimized.
def objective(trial):

    # Invoke suggest methods of a Trial object to generate hyperparameters.
    regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
    if regressor_name == 'SVR':
        svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
        regressor_obj = sklearn.svm.SVR(C=svr_c)
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
        regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)

    X, y = sklearn.datasets.fetch_california_housing(return_X_y=True)
    X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)

    regressor_obj.fit(X_train, y_train)
    y_pred = regressor_obj.predict(X_val)

    error = sklearn.metrics.mean_squared_error(y_val, y_pred)

    return error  # An objective value linked with the Trial object.

study = optuna.create_study()  # Create a new study.
study.optimize(objective, n_trials=100)  # Invoke optimization of the objective function.

Web Dashboard

Optuna Dashboard is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importance, etc. in graphs and tables. You don’t need to create a Python script to call Optuna’s visualization functions. Feature requests and bug reports are welcome!

https://user-images.githubusercontent.com/5564044/204975098-95c2cb8c-0fb5-4388-abc4-da32f56cb4e5.gif

optuna-dashboard can be installed via pip:

$ pip install optuna-dashboard

Tip

Please check out the getting started section of Optuna Dashboard’s official documentation.

Communication

GitHub Discussions for questions.
GitHub Issues for bug reports and feature requests.

Contribution

Any contributions to Optuna are welcome! When you send a pull request, please follow the contribution guide.

License

MIT License (see LICENSE).

Optuna uses the codes from SciPy and fdlibm projects (see Third-party License).

Reference

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).

Installation

Optuna supports Python 3.7 or newer.

We recommend to install Optuna via pip:

$ pip install optuna

You can also install the development version of Optuna from master branch of Git repository:

$ pip install git+https://github.com/optuna/optuna.git

You can also install Optuna via conda:

$ conda install -c conda-forge optuna

Tutorial

If you are new to Optuna or want a general introduction, we highly recommend the below video.

Key Features

Showcases Optuna’s Key Features.

Lightweight, versatile, and platform agnostic architecture
Pythonic Search Space
Efficient Optimization Algorithms
Easy Parallelization
Quick Visualization for Hyperparameter Optimization Analysis

Recipes

Showcases the recipes that might help you using Optuna with comfort.

Saving/Resuming Study with RDB Backend
Multi-objective Optimization with Optuna
User Attributes
Command-Line Interface
User-Defined Sampler
User-Defined Pruner
Callback for Study.optimize
Specify Hyperparameters Manually
Ask-and-Tell Interface
Re-use the best trial
(File-based) Journal Storage
Human-in-the-loop Optimization with Optuna Dashboard
Optuna Artifacts Tutorial
Early-stopping independent evaluations by Wilcoxon pruner

Gallery generated by Sphinx-Gallery

API Reference

optuna

The optuna module is primarily used as an alias for basic Optuna functionality coded in other modules. Currently, two modules are aliased: (1) from optuna.study, functions regarding the Study lifecycle, and (2) from optuna.exceptions, the TrialPruned Exception raised when a trial is pruned.

`optuna.create_study`	Create a new `Study`.
`optuna.load_study`	Load the existing `Study` that has the specified name.
`optuna.delete_study`	Delete a `Study` object.
`optuna.copy_study`	Copy study from one storage to another.
`optuna.get_all_study_names`	Get all study names stored in a specified storage.
`optuna.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.TrialPruned`	Exception for pruned trials.

optuna.artifacts

The artifacts module provides the way to manage artifacts (output files) in Optuna.

class optuna.artifacts.FileSystemArtifactStore(base_path)[source]

An artifact store for file systems.

Parameters:: base_path (str | Path) – The base path to a directory to store artifacts.

Example

import os

import optuna
from optuna.artifacts import FileSystemArtifactStore
from optuna.artifacts import upload_artifact


base_path = "./artifacts"
os.makedirs(base_path, exist_ok=True)
artifact_store = FileSystemArtifactStore(base_path=base_path)


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(trial, file_path, artifact_store)
    return ...

Note

Added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0.

class optuna.artifacts.Boto3ArtifactStore(bucket_name, client=None, *, avoid_buf_copy=False)[source]

An artifact backend for Boto3.

Parameters:

bucket_name (str) – The name of the bucket to store artifacts.
client (S3Client | None) – A Boto3 client to use for storage operations. If not specified, a new client will be created.
avoid_buf_copy (bool) – If True, skip procedure to copy the content of the source file object to a buffer before uploading it to S3 ins. This is default to False because using upload_fileobj() method of Boto3 client might close the source file object.

Example

import optuna
from optuna.artifacts import upload_artifact
from optuna.artifacts import Boto3ArtifactStore


artifact_store = Boto3ArtifactStore("my-bucket")


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(trial, file_path, artifact_store)
    return ...

Note

Added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0.

class optuna.artifacts.GCSArtifactStore(bucket_name, client=None)[source]

An artifact backend for Google Cloud Storage (GCS).

Parameters:

bucket_name (str) – The name of the bucket to store artifacts.
client (google.cloud.storage.Client | None) – A google-cloud-storage Client to use for storage operations. If not specified, a new client will be created with default settings.

Example

import optuna
from optuna.artifacts import GCSArtifactStore, upload_artifact


artifact_backend = GCSArtifactStore("my-bucket")


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(trial, file_path, artifact_backend)
    return ...

Before running this code, you will have to install gcloud and run

gcloud auth application-default login

so that the Cloud Storage library can automatically find the credential.

Note

Added in v3.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.4.0.

class optuna.artifacts.Backoff(backend, *, max_retries=10, multiplier=2, min_delay=0.1, max_delay=30)[source]

An artifact store’s middleware for exponential backoff.

Example

import optuna
from optuna.artifacts import upload_artifact
from optuna.artifacts import Boto3ArtifactStore
from optuna.artifacts import Backoff


artifact_store = Backoff(Boto3ArtifactStore("my-bucket"))


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(trial, file_path, artifact_store)
    return ...

Parameters:

backend (ArtifactStore)
max_retries (int)
multiplier (float)
min_delay (float)
max_delay (float)

optuna.artifacts.upload_artifact(study_or_trial, file_path, artifact_store, *, storage=None, mimetype=None, encoding=None)[source]

Upload an artifact to the artifact store.

Parameters:

study_or_trial (Trial | FrozenTrial | Study) – A Trial object, a FrozenTrial, or a Study object.
file_path (str) – A path to the file to be uploaded.
artifact_store (ArtifactStore) – An artifact store.
storage (BaseStorage | None) – A storage object. If trial is not a Trial object, this argument is required.
mimetype (str | None) – A MIME type of the artifact. If not specified, the MIME type is guessed from the file extension.
encoding (str | None) – An encoding of the artifact, which is suitable for use as a Content-Encoding header (e.g. gzip). If not specified, the encoding is guessed from the file extension.

Returns:

An artifact ID.

Return type:

str

Note

Added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0.

optuna.cli

The cli module implements Optuna’s command-line functionality.

For detail, please see the result of

$ optuna --help

See also

The Command-Line Interface tutorial provides use-cases with examples.

optuna.distributions

The distributions module defines various classes representing probability distributions, mainly used to suggest initial hyperparameter values for an optimization trial. Distribution classes inherit from a library-internal BaseDistribution, and is initialized with specific parameters, such as the low and high endpoints for a IntDistribution.

Optuna users should not use distribution classes directly, but instead use utility functions provided by Trial such as suggest_int().

`optuna.distributions.FloatDistribution`	A distribution on floats.
`optuna.distributions.IntDistribution`	A distribution on integers.
`optuna.distributions.UniformDistribution`	A uniform distribution in the linear domain.
`optuna.distributions.LogUniformDistribution`	A uniform distribution in the log domain.
`optuna.distributions.DiscreteUniformDistribution`	A discretized uniform distribution in the linear domain.
`optuna.distributions.IntUniformDistribution`	A uniform distribution on integers.
`optuna.distributions.IntLogUniformDistribution`	A uniform distribution on integers in the log domain.
`optuna.distributions.CategoricalDistribution`	A categorical distribution.
`optuna.distributions.distribution_to_json`	Serialize a distribution to JSON format.
`optuna.distributions.json_to_distribution`	Deserialize a distribution in JSON format.
`optuna.distributions.check_distribution_compatibility`	A function to check compatibility of two distributions.

optuna.exceptions

The exceptions module defines Optuna-specific exceptions deriving from a base OptunaError class. Of special importance for library users is the TrialPruned exception to be raised if optuna.trial.Trial.should_prune() returns True for a trial that should be pruned.

`optuna.exceptions.OptunaError`	Base class for Optuna specific errors.
`optuna.exceptions.TrialPruned`	Exception for pruned trials.
`optuna.exceptions.CLIUsageError`	Exception for CLI.
`optuna.exceptions.StorageInternalError`	Exception for storage operation.
`optuna.exceptions.DuplicatedStudyError`	Exception for a duplicated study name.

optuna.importance

The importance module provides functionality for evaluating hyperparameter importances based on completed trials in a given study. The utility function get_param_importances() takes a Study and optional evaluator as two of its inputs. The evaluator must derive from BaseImportanceEvaluator, and is initialized as a FanovaImportanceEvaluator by default when not passed in. Users implementing custom evaluators should refer to either FanovaImportanceEvaluator, MeanDecreaseImpurityImportanceEvaluator, or PedAnovaImportanceEvaluator as a guide, paying close attention to the format of the return value from the Evaluator’s evaluate function.

Note

FanovaImportanceEvaluator takes over 1 minute when given a study that contains 1000+ trials. We published optuna-fast-fanova library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. If n_trials is more than 10000, the Cython implementation takes more than a minute, so you can use PedAnovaImportanceEvaluator instead, enabling the evaluation to finish in a second.

`optuna.importance.get_param_importances`	Evaluate parameter importances based on completed trials in the given study.
`optuna.importance.FanovaImportanceEvaluator`	fANOVA importance evaluator.
`optuna.importance.MeanDecreaseImpurityImportanceEvaluator`	Mean Decrease Impurity (MDI) parameter importance evaluator.
`optuna.importance.PedAnovaImportanceEvaluator`	PED-ANOVA importance evaluator.

optuna.integration

The integration module contains classes used to integrate Optuna with external machine learning frameworks.

Note

Optuna’s integration modules for third-party libraries have started migrating from Optuna itself to a package called optuna-integration. Please check the repository and the documentation.

For most of the ML frameworks supported by Optuna, the corresponding Optuna integration class serves only to implement a callback object and functions, compliant with the framework’s specific callback API, to be called with each intermediate step in the model training. The functionality implemented in these callbacks across the different ML frameworks includes:

Reporting intermediate model scores back to the Optuna trial using optuna.trial.Trial.report(),
According to the results of optuna.trial.Trial.should_prune(), pruning the current model by raising optuna.TrialPruned(), and
Reporting intermediate Optuna data such as the current trial number back to the framework, as done in MLflowCallback.

For scikit-learn, an integrated OptunaSearchCV estimator is available that combines scikit-learn BaseEstimator functionality with access to a class-level Study object.

Dependencies of each integration

We summarize the necessary dependencies for each integration.

Integration	Dependencies
AllenNLP	allennlp, torch, psutil, jsonnet
BoTorch	botorch, gpytorch, torch
CatBoost	catboost
ChainerMN	chainermn
Chainer	chainer
pycma	cma
Dask	distributed
FastAI	fastai
Keras	keras
LightGBMTuner	lightgbm, scikit-learn
LightGBMPruningCallback	lightgbm
MLflow	mlflow
MXNet	mxnet
PyTorch Distributed	torch
PyTorch (Ignite)	pytorch-ignite
PyTorch (Lightning)	pytorch-lightning
SHAP	scikit-learn, shap
Scikit-learn	pandas, scipy, scikit-learn
SKorch	skorch
TensorBoard	tensorboard, tensorflow
TensorFlow	tensorflow, tensorflow-estimator
TensorFlow + Keras	tensorflow
Weights & Biases	wandb
XGBoost	xgboost

optuna.logging

The logging module implements logging using the Python logging package. Library users may be especially interested in setting verbosity levels using set_verbosity() to one of optuna.logging.CRITICAL (aka optuna.logging.FATAL), optuna.logging.ERROR, optuna.logging.WARNING (aka optuna.logging.WARN), optuna.logging.INFO, or optuna.logging.DEBUG.

`optuna.logging.get_verbosity`	Return the current level for the Optuna's root logger.
`optuna.logging.set_verbosity`	Set the level for the Optuna's root logger.
`optuna.logging.disable_default_handler`	Disable the default handler of the Optuna's root logger.
`optuna.logging.enable_default_handler`	Enable the default handler of the Optuna's root logger.
`optuna.logging.disable_propagation`	Disable propagation of the library log outputs.
`optuna.logging.enable_propagation`	Enable propagation of the library log outputs.

optuna.pruners

The pruners module defines a BasePruner class characterized by an abstract prune() method, which, for a given trial and its associated study, returns a boolean value representing whether the trial should be pruned. This determination is made based on stored intermediate values of the objective function, as previously reported for the trial using optuna.trial.Trial.report(). The remaining classes in this module represent child classes, inheriting from BasePruner, which implement different pruning strategies.

Warning

Currently pruners module is expected to be used only for single-objective optimization.

See also

Efficient Optimization Algorithms tutorial explains the concept of the pruner classes and a minimal example.

See also

User-Defined Pruner tutorial could be helpful if you want to implement your own pruner classes.

`optuna.pruners.BasePruner`	Base class for pruners.
`optuna.pruners.MedianPruner`	Pruner using the median stopping rule.
`optuna.pruners.NopPruner`	Pruner which never prunes trials.
`optuna.pruners.PatientPruner`	Pruner which wraps another pruner with tolerance.
`optuna.pruners.PercentilePruner`	Pruner to keep the specified percentile of the trials.
`optuna.pruners.SuccessiveHalvingPruner`	Pruner using Asynchronous Successive Halving Algorithm.
`optuna.pruners.HyperbandPruner`	Pruner using Hyperband.
`optuna.pruners.ThresholdPruner`	Pruner to detect outlying metrics of the trials.
`optuna.pruners.WilcoxonPruner`	Pruner based on the Wilcoxon signed-rank test.

optuna.samplers

The samplers module defines a base class for parameter sampling as described extensively in BaseSampler. The remaining classes in this module represent child classes, deriving from BaseSampler, which implement different sampling strategies.

See also

Efficient Optimization Algorithms tutorial explains the overview of the sampler classes.

See also

User-Defined Sampler tutorial could be helpful if you want to implement your own sampler classes.

	RandomSampler	GridSampler	TPESampler	CmaEsSampler	NSGAIISampler	QMCSampler	GPSampler	BoTorchSampler	BruteForceSampler
Float parameters	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\) (\(\color{red}\times\) for infinite domain)
Integer parameters	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)
Categorical parameters	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)
Pruning	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{red}\times\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\color{green}\checkmark\)
Multivariate optimization	\(\blacktriangle\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)
Conditional search space	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\blacktriangle\)	\(\color{green}\checkmark\)
Multi-objective optimization	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{red}\times\)	\(\color{green}\checkmark\) (\(\blacktriangle\) for single-objective)	\(\color{green}\checkmark\)	\(\color{red}\times\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)
Batch optimization	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)
Distributed optimization	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)	\(\blacktriangle\)	\(\color{green}\checkmark\)	\(\color{green}\checkmark\)
Constrained optimization	\(\color{red}\times\)	\(\color{red}\times\)	\(\color{green}\checkmark\)	\(\color{red}\times\)	\(\color{green}\checkmark\)	\(\color{red}\times\)	\(\color{red}\times\)	\(\color{green}\checkmark\)	\(\color{red}\times\)
Time complexity (per trial) (*)	\(O(d)\)	\(O(dn)\)	\(O(dn \log n)\)	\(O(d^3)\)	\(O(mp^2)\) (***)	\(O(dn)\)	\(O(n^3)\)	\(O(n^3)\)	\(O(d)\)
Recommended budgets (#trials) (**)	as many as one likes	number of combinations	100 – 1000	1000 – 10000	100 – 10000	as many as one likes	– 500	10 – 100	number of combinations

Note

\(\color{green}\checkmark\): Supports this feature. \(\blacktriangle\): Works, but inefficiently. \(\color{red}\times\): Causes an error, or has no interface.

(*): We assumes that \(d\) is the dimension of the search space, \(n\) is the number of finished trials, \(m\) is the number of objectives, and \(p\) is the population size (algorithm specific parameter). This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including \(O(d)\) to call the sampling methods and \(O(n)\) to collect the completed trials. This means that, for example, the actual time complexity of RandomSampler is \(O(d+n+d) = O(d+n)\). From another perspective, with the exception of NSGAIISampler, all time complexity is written for single-objective optimization.

(**): (1) The budget depends on the number of parameters and the number of objectives. (2) This budget includes n_startup_trials if a sampler has n_startup_trials as one of its arguments.

(***): This time complexity assumes that the number of population size \(p\) and the number of parallelization are regular. This means that the number of parallelization should not exceed the number of population size \(p\).

Note

Samplers initialize their random number generators by specifying seed argument at initialization. However, samplers reseed them when n_jobs!=1 of optuna.study.Study.optimize() to avoid sampling duplicated parameters by using the same generator. Thus we can hardly reproduce the optimization results with n_jobs!=1. For the same reason, make sure that use either seed=None or different seed values among processes with distributed optimization explained in Easy Parallelization tutorial.

Note

For float, integer, or categorical parameters, see Pythonic Search Space tutorial.

For pruning, see Efficient Optimization Algorithms tutorial.

For multivariate optimization, see BaseSampler. The multivariate optimization is implemented as sample_relative() in Optuna. Please check the concrete documents of samplers for more details.

For conditional search space, see Pythonic Search Space tutorial and TPESampler. The group option of TPESampler allows TPESampler to handle the conditional search space.

For multi-objective optimization, see Multi-objective Optimization with Optuna tutorial.

For batch optimization, see Batch Optimization tutorial. Note that the constant_liar option of TPESampler allows TPESampler to handle the batch optimization.

For distributed optimization, see Easy Parallelization tutorial. Note that the constant_liar option of TPESampler allows TPESampler to handle the distributed optimization.

For constrained optimization, see an example.

`optuna.samplers.BaseSampler`	Base class for samplers.
`optuna.samplers.GridSampler`	Sampler using grid search.
`optuna.samplers.RandomSampler`	Sampler using random sampling.
`optuna.samplers.TPESampler`	Sampler using TPE (Tree-structured Parzen Estimator) algorithm.
`optuna.samplers.CmaEsSampler`	A sampler using cmaes as the backend.
`optuna.samplers.GPSampler`	Sampler using Gaussian process-based Bayesian optimization.
`optuna.samplers.PartialFixedSampler`	Sampler with partially fixed parameters.
`optuna.samplers.NSGAIISampler`	Multi-objective sampler using the NSGA-II algorithm.
`optuna.samplers.NSGAIIISampler`	Multi-objective sampler using the NSGA-III algorithm.
`optuna.samplers.QMCSampler`	A Quasi Monte Carlo Sampler that generates low-discrepancy sequences.
`optuna.samplers.BruteForceSampler`	Sampler using brute force.

Note

The following optuna.samplers.nsgaii module defines crossover operations used by NSGAIISampler.

optuna.samplers.nsgaii

The nsgaii module defines crossover operations used by NSGAIISampler.

`optuna.samplers.nsgaii.BaseCrossover`	Base class for crossovers.
`optuna.samplers.nsgaii.UniformCrossover`	Uniform Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.BLXAlphaCrossover`	Blend Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.SPXCrossover`	Simplex Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.SBXCrossover`	Simulated Binary Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.VSBXCrossover`	Modified Simulated Binary Crossover operation used by `NSGAIISampler`.
`optuna.samplers.nsgaii.UNDXCrossover`	Unimodal Normal Distribution Crossover used by `NSGAIISampler`.

optuna.search_space

The search_space module provides functionality for controlling search space of parameters.

`optuna.search_space.IntersectionSearchSpace`	A class to calculate the intersection search space of a `Study`.
`optuna.search_space.intersection_search_space`	Return the intersection search space of the given trials.

optuna.storages

The storages module defines a BaseStorage class which abstracts a backend database and provides library-internal interfaces to the read/write histories of the studies and trials. Library users who wish to use storage solutions other than the default in-memory storage should use one of the child classes of BaseStorage documented below.

`optuna.storages.RDBStorage`	Storage class for RDB backend.
`optuna.storages.RetryFailedTrialCallback`	Retry a failed trial up to a maximum number of times.
`optuna.storages.fail_stale_trials`	Fail stale trials and run their failure callbacks.
`optuna.storages.JournalStorage`	Storage class for Journal storage backend.
`optuna.storages.JournalFileStorage`	File storage class for Journal log backend.
`optuna.storages.JournalFileSymlinkLock`	Lock class for synchronizing processes for NFSv2 or later.
`optuna.storages.JournalFileOpenLock`	Lock class for synchronizing processes for NFSv3 or later.
`optuna.storages.JournalRedisStorage`	Redis storage class for Journal log backend.

optuna.study

The study module implements the Study object and related functions. A public constructor is available for the Study class, but direct use of this constructor is not recommended. Instead, library users should create and load a Study using create_study() and load_study() respectively.

`optuna.study.Study`	A study corresponds to an optimization task, i.e., a set of trials.
`optuna.study.create_study`	Create a new `Study`.
`optuna.study.load_study`	Load the existing `Study` that has the specified name.
`optuna.study.delete_study`	Delete a `Study` object.
`optuna.study.copy_study`	Copy study from one storage to another.
`optuna.study.get_all_study_names`	Get all study names stored in a specified storage.
`optuna.study.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.study.MaxTrialsCallback`	Set a maximum number of trials before ending the study.
`optuna.study.StudyDirection`	Direction of a `Study`.
`optuna.study.StudySummary`	Basic attributes and aggregated results of a `Study`.

optuna.terminator

The terminator module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator stops the optimization process when the estimated potential improvement is smaller than the statistical error.

`optuna.terminator.BaseTerminator`	Base class for terminators.
`optuna.terminator.Terminator`	Automatic stopping mechanism for Optuna studies.
`optuna.terminator.BaseImprovementEvaluator`	Base class for improvement evaluators.
`optuna.terminator.RegretBoundEvaluator`	An error evaluator for upper bound on the regret with high-probability confidence.
`optuna.terminator.BestValueStagnationEvaluator`	Evaluates the stagnation period of the best value in an optimization process.
`optuna.terminator.BaseErrorEvaluator`	Base class for error evaluators.
`optuna.terminator.CrossValidationErrorEvaluator`	An error evaluator for objective functions based on cross-validation.
`optuna.terminator.StaticErrorEvaluator`	An error evaluator that always returns a constant value.
`optuna.terminator.TerminatorCallback`	A callback that terminates the optimization using Terminator.
`optuna.terminator.report_cross_validation_scores`	A function to report cross-validation scores of a trial.

For an example of using this module, please refer to this example.

optuna.trial

The trial module contains Trial related classes and functions.

A Trial instance represents a process of evaluating an objective function. This instance is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial’s state, and set/get user-defined attributes of the trial, so that Optuna users can define a custom objective function through the interfaces. Basically, Optuna users only use it in their custom objective functions.

`optuna.trial.Trial`	A trial is a process of evaluating an objective function.
`optuna.trial.FixedTrial`	A trial class which suggests a fixed value for each parameter.
`optuna.trial.FrozenTrial`	Status and results of a `Trial`.
`optuna.trial.TrialState`	State of a `Trial`.
`optuna.trial.create_trial`	Create a new `FrozenTrial`.

optuna.visualization

The visualization module provides utility functions for plotting the optimization process using plotly and matplotlib. Plotting functions generally take a Study object and optional parameters are passed as a list to the params argument.

Note

In the optuna.visualization module, the following functions use plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

Note

The plot_param_importances() requires the Python package of scikit-learn.

`optuna.visualization.plot_contour`	Plot the parameter relationship as contour plot in a study.
`optuna.visualization.plot_edf`	Plot the objective value EDF (empirical distribution function) of a study.
`optuna.visualization.plot_hypervolume_history`	Plot hypervolume history of all trials in a study.
`optuna.visualization.plot_intermediate_values`	Plot intermediate values of all trials in a study.
`optuna.visualization.plot_optimization_history`	Plot optimization history of all trials in a study.
`optuna.visualization.plot_parallel_coordinate`	Plot the high-dimensional parameter relationships in a study.
`optuna.visualization.plot_param_importances`	Plot hyperparameter importances.
`optuna.visualization.plot_pareto_front`	Plot the Pareto front of a study.
`optuna.visualization.plot_rank`	Plot parameter relations as scatter plots with colors indicating ranks of target value.
`optuna.visualization.plot_slice`	Plot the parameter relationship as slice plot in a study.
`optuna.visualization.plot_terminator_improvement`	Plot the potentials for future objective improvement.
`optuna.visualization.plot_timeline`	Plot the timeline of a study.
`optuna.visualization.is_available`	Returns whether visualization with plotly is available or not.

Note

The following optuna.visualization.matplotlib module uses Matplotlib as a backend.

optuna.visualization.matplotlib

Note

The following functions use Matplotlib as a backend.

`optuna.visualization.matplotlib.plot_contour`	Plot the parameter relationship as contour plot in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_edf`	Plot the objective value EDF (empirical distribution function) of a study with Matplotlib.
`optuna.visualization.matplotlib.plot_hypervolume_history`	Plot hypervolume history of all trials in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_intermediate_values`	Plot intermediate values of all trials in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_optimization_history`	Plot optimization history of all trials in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_parallel_coordinate`	Plot the high-dimensional parameter relationships in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_param_importances`	Plot hyperparameter importances with Matplotlib.
`optuna.visualization.matplotlib.plot_pareto_front`	Plot the Pareto front of a study.
`optuna.visualization.matplotlib.plot_rank`	Plot parameter relations as scatter plots with colors indicating ranks of target value.
`optuna.visualization.matplotlib.plot_slice`	Plot the parameter relationship as slice plot in a study with Matplotlib.
`optuna.visualization.matplotlib.plot_terminator_improvement`	Plot the potentials for future objective improvement.
`optuna.visualization.matplotlib.plot_timeline`	Plot the timeline of a study.
`optuna.visualization.matplotlib.is_available`	Returns whether visualization with Matplotlib is available or not.

See also

The Quick Visualization for Hyperparameter Optimization Analysis tutorial provides use-cases with examples.

FAQ

Can I use Optuna with X? (where X is your favorite ML library)

Optuna is compatible with most ML libraries, and it’s easy to use Optuna with those. Please refer to examples.

How to define objective functions that have own arguments?

There are two ways to realize it.

First, callable classes can be used for that purpose as follows:

import optuna


class Objective:
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_float("x", self.min_x, self.max_x)
        return (x - 2) ** 2


# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

Second, you can use lambda or functools.partial for creating functions (closures) that hold extra arguments. Below is an example that uses lambda:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_float("x", min_x, max_x)
    return (x - 2) ** 2


# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

Please also refer to sklearn_additional_args.py example, which reuses the dataset instead of loading it in each trial execution.

Can I use Optuna without remote RDB servers?

Yes, it’s possible.

In the simplest form, Optuna works with in-memory storage:

study = optuna.create_study()
study.optimize(objective)

If you want to save and resume studies, it’s handy to use SQLite as the local storage:

study = optuna.create_study(study_name="foo_study", storage="sqlite:///example.db")
study.optimize(objective)  # The state of `study` will be persisted to the local SQLite file.

Please see Saving/Resuming Study with RDB Backend for more details.

How can I save and resume studies?

There are two ways of persisting studies, which depend if you are using in-memory storage (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using pickle or joblib. For example, using joblib:

study = optuna.create_study()
joblib.dump(study, "study.pkl")

And to resume the study:

study = joblib.load("study.pkl")
print("Best trial until now:")
print(" Value: ", study.best_trial.value)
print(" Params: ")
for key, value in study.best_trial.params.items():
    print(f"    {key}: {value}")

Note that Optuna does not support saving/reloading across different Optuna versions with pickle. To save/reload a study across different Optuna versions, please use RDBs and upgrade storage schema if necessary. If you are using RDBs, see Saving/Resuming Study with RDB Backend for more details.

How to suppress log messages of Optuna?

By default, Optuna shows log messages at the optuna.logging.INFO level. You can change logging levels by using optuna.logging.set_verbosity().

For instance, you can stop showing each trial result as follows:

optuna.logging.set_verbosity(optuna.logging.WARNING)

study = optuna.create_study()
study.optimize(objective)
# Logs like '[I 2020-07-21 13:41:45,627] Trial 0 finished with value:...' are disabled.

Please refer to optuna.logging for further details.

How to save machine learning models trained in objective functions?

Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used.

We recommend saving optuna.trial.Trial.number with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows:

def objective(trial):
    svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
    clf = sklearn.svm.SVC(C=svc_c)
    clf.fit(X_train, y_train)

    # Save a trained model to a file.
    with open("{}.pickle".format(trial.number), "wb") as fout:
        pickle.dump(clf, fout)
    return 1.0 - accuracy_score(y_valid, clf.predict(X_valid))


study = optuna.create_study()
study.optimize(objective, n_trials=100)

# Load the best model.
with open("{}.pickle".format(study.best_trial.number), "rb") as fin:
    best_clf = pickle.load(fin)
print(accuracy_score(y_valid, best_clf.predict(X_valid)))

How can I obtain reproducible optimization results?

To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via seed argument of an instance of samplers as follows:

sampler = TPESampler(seed=10)  # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)
study.optimize(objective)

However, there are two caveats.

First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result.

Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it.

How are exceptions from trials handled?

Trials that raise exceptions without catching them will be treated as failures, i.e. with the FAIL status.

By default, all exceptions except TrialPruned raised in objective functions are propagated to the caller of optimize(). In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in optimize() which exception types to catch using the catch argument. Exceptions of these types are caught inside the study and will not propagate further.

You can find the failed trials in log messages.

[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')

You can also find the failed trials by checking the trial states as follows:

study.trials_dataframe()

number	state	value	…	params	system_attrs
0	TrialState.FAIL		…	0	Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’)
1	TrialState.COMPLETE	1269	…	1

See also

The catch argument in optimize().

How are NaNs returned by trials handled?

Trials that return NaN (float('nan')) are treated as failures, but they will not abort studies.

Trials which return NaN are shown as follows:

[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.

What happens when I dynamically alter a search space?

Since parameters search spaces are specified in each call to the suggestion API, e.g. suggest_float() and suggest_int(), it is possible to, in a single study, alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually.

Note

Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822

How can I use two GPUs for evaluating two trials simultaneously?

If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used in your script, main.py, the easiest way is to set CUDA_VISIBLE_DEVICES environment variable:

# On a terminal.
#
# Specify to use the first GPU, and run an optimization.
$ export CUDA_VISIBLE_DEVICES=0
$ python main.py

# On another terminal.
#
# Specify to use the second GPU, and run another optimization.
$ export CUDA_VISIBLE_DEVICES=1
$ python main.py

Please refer to CUDA C Programming Guide for further details.

How can I test my objective functions?

When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use FixedTrial, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of \(x\) and \(y\) to the objective function \(x + y\) as follows:

def objective(trial):
    x = trial.suggest_float("x", -1.0, 1.0)
    y = trial.suggest_int("y", -5, 5)
    return x + y


objective(FixedTrial({"x": 1.0, "y": -1}))  # 0.0
objective(FixedTrial({"x": -1.0, "y": -4}))  # -5.0

Using FixedTrial, you can write unit tests as follows:

# A test function of pytest
def test_objective():
    assert 1.0 == objective(FixedTrial({"x": 1.0, "y": 0}))
    assert -1.0 == objective(FixedTrial({"x": 0.0, "y": -1}))
    assert 0.0 == objective(FixedTrial({"x": -1.0, "y": 1}))

How do I avoid running out of memory (OOM) when optimizing studies?

If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify gc_after_trial to True when calling optimize() or call gc.collect() inside a callback.

def objective(trial):
    x = trial.suggest_float("x", -1.0, 1.0)
    y = trial.suggest_int("y", -5, 5)
    return x + y


study = optuna.create_study()
study.optimize(objective, n_trials=10, gc_after_trial=True)

# `gc_after_trial=True` is more or less identical to the following.
study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()])

There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, gc_after_trial is False by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that gc.collect() is called even when errors, including TrialPruned are raised.

Note

ChainerMNStudy does currently not provide gc_after_trial nor callbacks for optimize(). When using this class, you will have to call the garbage collector inside the objective function.

How can I output a log only when the best value is updated?

Here’s how to replace the logging feature of optuna with your own logging callback function. The implemented callback can be passed to optimize(). Here’s an example:

import optuna


# Turn off optuna log notes.
optuna.logging.set_verbosity(optuna.logging.WARN)


def objective(trial):
    x = trial.suggest_float("x", 0, 1)
    return x ** 2


def logging_callback(study, frozen_trial):
    previous_best_value = study.user_attrs.get("previous_best_value", None)
    if previous_best_value != study.best_value:
        study.set_user_attr("previous_best_value", study.best_value)
        print(
            "Trial {} finished with best value: {} and parameters: {}. ".format(
            frozen_trial.number,
            frozen_trial.value,
            frozen_trial.params,
            )
        )


study = optuna.create_study()
study.optimize(objective, n_trials=100, callbacks=[logging_callback])

Note that this callback may show incorrect values when you try to optimize an objective function with n_jobs!=1 (or other forms of distributed optimization) due to its reads and writes to storage that are prone to race conditions.

How do I suggest variables which represent the proportion, that is, are in accordance with Dirichlet distribution?

When you want to suggest \(n\) variables which represent the proportion, that is, \(p[0], p[1], ..., p[n-1]\) which satisfy \(0 \le p[k] \le 1\) for any \(k\) and \(p[0] + p[1] + ... + p[n-1] = 1\), try the below. For example, these variables can be used as weights when interpolating the loss functions. These variables are in accordance with the flat Dirichlet distribution.

import numpy as np
import matplotlib.pyplot as plt
import optuna


def objective(trial):
    n = 5
    x = []
    for i in range(n):
        x.append(- np.log(trial.suggest_float(f"x_{i}", 0, 1)))

    p = []
    for i in range(n):
        p.append(x[i] / sum(x))

    for i in range(n):
        trial.set_user_attr(f"p_{i}", p[i])

    return 0

study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
study.optimize(objective, n_trials=1000)

n = 5
p = []
for i in range(n):
    p.append([trial.user_attrs[f"p_{i}"] for trial in study.trials])
axes = plt.subplots(n, n, figsize=(20, 20))[1]

for i in range(n):
    for j in range(n):
        axes[j][i].scatter(p[i], p[j], marker=".")
        axes[j][i].set_xlim(0, 1)
        axes[j][i].set_ylim(0, 1)
        axes[j][i].set_xlabel(f"p_{i}")
        axes[j][i].set_ylabel(f"p_{j}")

plt.savefig("sampled_ps.png")

This method is justified in the following way: First, if we apply the transformation \(x = - \log (u)\) to the variable \(u\) sampled from the uniform distribution \(Uni(0, 1)\) in the interval \([0, 1]\), the variable \(x\) will follow the exponential distribution \(Exp(1)\) with scale parameter \(1\). Furthermore, for \(n\) variables \(x[0], ..., x[n-1]\) that follow the exponential distribution of scale parameter \(1\) independently, normalizing them with \(p[i] = x[i] / \sum_i x[i]\), the vector \(p\) follows the Dirichlet distribution \(Dir(\alpha)\) of scale parameter \(\alpha = (1, ..., 1)\). You can verify the transformation by calculating the elements of the Jacobian.

How can I optimize a model with some constraints?

When you want to optimize a model with constraints, you can use the following classes: TPESampler, NSGAIISampler or BoTorchSampler. The following example is a benchmark of Binh and Korn function, a multi-objective optimization, with constraints using NSGAIISampler. This one has two constraints \(c_0 = (x-5)^2 + y^2 - 25 \le 0\) and \(c_1 = -(x - 8)^2 - (y + 3)^2 + 7.7 \le 0\) and finds the optimal solution satisfying these constraints.

import optuna


def objective(trial):
    # Binh and Korn function with constraints.
    x = trial.suggest_float("x", -15, 30)
    y = trial.suggest_float("y", -15, 30)

    # Constraints which are considered feasible if less than or equal to zero.
    # The feasible region is basically the intersection of a circle centered at (x=5, y=0)
    # and the complement to a circle centered at (x=8, y=-3).
    c0 = (x - 5) ** 2 + y ** 2 - 25
    c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7

    # Store the constraints as user attributes so that they can be restored after optimization.
    trial.set_user_attr("constraint", (c0, c1))

    v0 = 4 * x ** 2 + 4 * y ** 2
    v1 = (x - 5) ** 2 + (y - 5) ** 2

    return v0, v1


def constraints(trial):
    return trial.user_attrs["constraint"]


sampler = optuna.samplers.NSGAIISampler(constraints_func=constraints)
study = optuna.create_study(
    directions=["minimize", "minimize"],
    sampler=sampler,
)
study.optimize(objective, n_trials=32, timeout=600)

print("Number of finished trials: ", len(study.trials))

print("Pareto front:")

trials = sorted(study.best_trials, key=lambda t: t.values)

for trial in trials:
    print("  Trial#{}".format(trial.number))
    print(
        "    Values: Values={}, Constraint={}".format(
            trial.values, trial.user_attrs["constraint"][0]
        )
    )
    print("    Params: {}".format(trial.params))

If you are interested in an example for BoTorchSampler, please refer to this sample code.

There are two kinds of constrained optimizations, one with soft constraints and the other with hard constraints. Soft constraints do not have to be satisfied, but an objective function is penalized if they are unsatisfied. On the other hand, hard constraints must be satisfied.

Optuna is adopting the soft one and DOES NOT support the hard one. In other words, Optuna DOES NOT have built-in samplers for the hard constraints.

How can I parallelize optimization?

The variations of parallelization are in the following three cases.

Multi-threading parallelization with single node
Multi-processing parallelization with single node
Multi-processing parallelization with multiple nodes

1. Multi-threading parallelization with a single node 

Parallelization can be achieved by setting the argument n_jobs in optuna.study.Study.optimize(). However, the python code will not be faster due to GIL because optuna.study.Study.optimize() with n_jobs!=1 uses multi-threading.

While optimizing, it will be faster in limited situations, such as waiting for other server requests or C/C++ processing with numpy, etc., but it will not be faster in other cases.

For more information about 1., see APIReference.

2. Multi-processing parallelization with single node 

This can be achieved by using JournalFileStorage or client/server RDBs (such as PostgreSQL and MySQL).

For more information about 2., see TutorialEasyParallelization.

3. Multi-processing parallelization with multiple nodes 

This can be achieved by using client/server RDBs (such as PostgreSQL and MySQL). However, if you are in the environment where you can not install a client/server RDB, you can not run multi-processing parallelization with multiple nodes.

For more information about 3., see TutorialEasyParallelization.

How can I solve the error that occurs when performing parallel optimization with SQLite3?

We would never recommend SQLite3 for parallel optimization in the following reasons.

To concurrently evaluate trials enqueued by enqueue_trial(), RDBStorage uses SELECT … FOR UPDATE syntax, which is unsupported in SQLite3.
As described in the SQLAlchemy’s documentation, SQLite3 (and pysqlite driver) does not support a high level of concurrency. You may get a “database is locked” error, which occurs when one thread or process has an exclusive lock on a database connection (in reality a file handle) and another thread times out waiting for the lock to be released. You can increase the default timeout value like optuna.storages.RDBStorage(“sqlite:///example.db”, engine_kwargs={“connect_args”: {“timeout”: 20.0}}) though.
For distributed optimization via NFS, SQLite3 does not work as described at FAQ section of sqlite.org.

If you want to use a file-based Optuna storage for these scenarios, please consider using JournalFileStorage instead.

import optuna
from optuna.storages import JournalStorage, JournalFileStorage

storage = JournalStorage(JournalFileStorage("optuna-journal.log"))
study = optuna.create_study(storage=storage)
...

See the Medium blog post for details.

Can I monitor trials and make them failed automatically when they are killed unexpectedly?

Note

Heartbeat mechanism is experimental. API would change in the future.

A process running a trial could be killed unexpectedly, typically by a job scheduler in a cluster environment. If trials are killed unexpectedly, they will be left on the storage with their states RUNNING until we remove them or update their state manually. For such a case, Optuna supports monitoring trials using heartbeat mechanism. Using heartbeat, if a process running a trial is killed unexpectedly, Optuna will automatically change the state of the trial that was running on that process to FAIL from RUNNING.

import optuna

def objective(trial):
    (Very time-consuming computation)

# Recording heartbeats every 60 seconds.
# Other processes' trials where more than 120 seconds have passed
# since the last heartbeat was recorded will be automatically failed.
storage = optuna.storages.RDBStorage(url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120)
study = optuna.create_study(storage=storage)
study.optimize(objective, n_trials=100)

Note

The heartbeat is supposed to be used with optimize(). If you use ask() and tell(), please change the state of the killed trials by calling tell() explicitly.

You can also execute a callback function to process the failed trial. Optuna provides a callback to retry failed trials as RetryFailedTrialCallback. Note that a callback is invoked at a beginning of each trial, which means RetryFailedTrialCallback will retry failed trials when a new trial starts to evaluate.

import optuna
from optuna.storages import RetryFailedTrialCallback

storage = optuna.storages.RDBStorage(
    url="sqlite:///:memory:",
    heartbeat_interval=60,
    grace_period=120,
    failed_trial_callback=RetryFailedTrialCallback(max_retry=3),
)

study = optuna.create_study(storage=storage)

How can I deal with permutation as a parameter?

Although it is not straightforward to deal with combinatorial search spaces like permutations with existing API, there exists a convenient technique for handling them. It involves re-parametrization of permutation search space of \(n\) items as an independent \(n\)-dimensional integer search space. This technique is based on the concept of Lehmer code.

A Lehmer code of a sequence is the sequence of integers in the same size, whose \(i\)-th entry denotes how many inversions the \(i\)-th entry of the permutation has after itself. In other words, the \(i\)-th entry of the Lehmer code represents the number of entries that are located after and are smaller than the \(i\)-th entry of the original sequence. For instance, the Lehmer code of the permutation \((3, 1, 4, 2, 0)\) is \((3, 1, 2, 1, 0)\).

Not only does the Lehmer code provide a unique encoding of permutations into an integer space, but it also has some desirable properties. For example, the sum of Lehmer code entries is equal to the minimum number of adjacent transpositions necessary to transform the corresponding permutation into the identity permutation. Additionally, the lexicographical order of the encodings of two permutations is the same as that of the original sequence. Therefore, Lehmer code preserves “closeness” among permutations in some sense, which is important for the optimization algorithm. An Optuna implementation example to solve Euclid TSP is as follows:

import numpy as np

import optuna


def decode(lehmer_code: list[int]) -> list[int]:
    """Decode Lehmer code to permutation.

    This function decodes Lehmer code represented as a list of integers to a permutation.
    """
    all_indices = list(range(n))
    output = []
    for k in lehmer_code:
        value = all_indices[k]
        output.append(value)
        all_indices.remove(value)
    return output


# Euclidean coordinates of cities for TSP.
city_coordinates = np.array(
    [[0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [2.0, 2.0], [-1.0, -1.0]]
)
n = len(city_coordinates)


def objective(trial: optuna.Trial) -> float:
    # Suggest a permutation in the Lehmer code representation.
    lehmer_code = [trial.suggest_int(f"x{i}", 0, n - i - 1) for i in range(n)]
    permutation = decode(lehmer_code)

    # Calculate the total distance of the suggested path.
    total_distance = 0.0
    for i in range(n):
        total_distance += np.linalg.norm(
            city_coordinates[permutation[i]] - city_coordinates[np.roll(permutation, 1)[i]]
        )
    return total_distance


study = optuna.create_study()
study.optimize(objective, n_trials=10)
lehmer_code = study.best_params.values()
print(decode(lehmer_code))

How can I ignore duplicated samples?

Optuna may sometimes suggest parameters evaluated in the past and if you would like to avoid this problem, you can try out the following workaround:

import optuna
from optuna.trial import TrialState


def objective(trial):
    # Sample parameters.
    x = trial.suggest_int("x", -5, 5)
    y = trial.suggest_int("y", -5, 5)
    # Fetch all the trials to consider.
    # In this example, we use only completed trials, but users can specify other states
    # such as TrialState.PRUNED and TrialState.FAIL.
    states_to_consider = (TrialState.COMPLETE,)
    trials_to_consider = trial.study.get_trials(deepcopy=False, states=states_to_consider)
    # Check whether we already evaluated the sampled `(x, y)`.
    for t in reversed(trials_to_consider):
        if trial.params == t.params:
            # Use the existing value as trial duplicated the parameters.
            return t.value

    # Compute the objective function if the parameters are not duplicated.
    # We use the 2D sphere function in this example.
    return x ** 2 + y ** 2


study = optuna.create_study()
study.optimize(objective, n_trials=100)

Optuna: A hyperparameter optimization framework

Key Features

Basic Concepts

Web Dashboard

Communication

Contribution

License

Reference

Installation

Tutorial

Key Features

Recipes

API Reference

optuna

optuna.artifacts

optuna.cli

optuna.distributions

optuna.exceptions

optuna.importance

optuna.integration

Dependencies of each integration

optuna.logging

optuna.pruners

optuna.samplers

optuna.samplers.nsgaii

optuna.search_space

optuna.storages

optuna.study

optuna.terminator

optuna.trial

optuna.visualization

optuna.visualization.matplotlib

FAQ

Indices and tables